{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {},
  "cells": [
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "   <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "\n",
        "<td>\n",
        "<a href=\"https://colab.research.google.com/github/Labelbox/labelbox-python/blob/master/examples/prediction_upload/image_predictions.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
        "</td>\n",
        "\n",
        "<td>\n",
        "<a href=\"https://github.com/Labelbox/labelbox-python/blob/master/examples/prediction_upload/image_predictions.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Image Prediction Import\n",
        "\n",
        "* This notebook walks you through the process of uploading model predictions to a Model Run. This notebook provides an example for each supported prediction type for image assets. \n",
        "\n",
        "A Model Run is a container for the predictions, annotations and metrics of a specific experiment in your ML model development cycle.\n",
        "\n",
        "**Supported annotations that can be uploaded through the SDK**\n",
        "\n",
        "- Bounding box \n",
        "- Polygon\n",
        "- Point\n",
        "- Polyline \n",
        "- Raster Segmentation\n",
        "- Classification free-text\n",
        "- Classification - radio\n",
        "- Classification - checklist\n",
        "\n",
        "\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "* Notes:\n",
        "    * If you are importing more than 1,000 mask predictions at a time, consider submitting separate jobs, as they can take longer than other prediction types to import.\n",
        "    * After the execution of this notebook a complete Model Run with predictions will be created in your organization. "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "## Setup"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "!pip install -q \"labelbox[data]\""
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "import uuid\n",
        "import requests\n",
        "import labelbox as lb\n",
        "import labelbox.types as lb_types"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Replace with your API Key \n",
        "Guides on [Create an API key](https://docs.labelbox.com/docs/create-an-api-key)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "API_KEY = \"\"\n",
        "client = lb.Client(API_KEY)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Supported Predictions"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "### Classification: Radio (single-choice)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Python annotation\n",
        "radio_prediction = lb_types.ClassificationAnnotation(\n",
        "    name=\"radio_question\",\n",
        "    value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = \"second_radio_answer\", confidence=0.5))\n",
        ")\n",
        "\n",
        "# NDJSON\n",
        "radio_prediction_ndjson = {\n",
        "  \"name\": \"radio_question\",\n",
        "  \"answer\": {\"name\": \"second_radio_answer\", \"confidence\": 0.5}\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Classification: Nested radio and checklist"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "nested_radio_prediction = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_radio_question\",\n",
        "  value=lb_types.Radio(\n",
        "    answer=lb_types.ClassificationAnswer(\n",
        "      name=\"first_radio_answer\",\n",
        "      confidence=0.5,\n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_radio_question\",\n",
        "          value=lb_types.Radio(\n",
        "            answer=lb_types.ClassificationAnswer(\n",
        "              name=\"first_sub_radio_answer\",\n",
        "              confidence=0.5\n",
        "            )\n",
        "          )\n",
        "        )\n",
        "      ]\n",
        "    )\n",
        "  )\n",
        ")\n",
        "\n",
        "nested_radio_prediction_ndjson = {\n",
        "    \"name\": \"nested_radio_question\",\n",
        "    \"answer\": {\n",
        "        \"name\": \"first_radio_answer\",\n",
        "        \"confidence\": 0.5,\n",
        "        \"classifications\": [{\n",
        "            \"name\": \"sub_radio_question\",\n",
        "            \"answer\": {\n",
        "                \"name\": \"first_sub_radio_answer\",\n",
        "                \"confidence\": 0.5\n",
        "            }\n",
        "        }]\n",
        "    }\n",
        "}\n",
        "\n",
        "\n",
        "nested_checklist_prediction = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_checklist_question\",\n",
        "  value=lb_types.Checklist(\n",
        "    answer=[lb_types.ClassificationAnswer(\n",
        "      name=\"first_checklist_answer\",\n",
        "      confidence=0.5,\n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_checklist_question\",\n",
        "          value=lb_types.Checklist(\n",
        "            answer=[lb_types.ClassificationAnswer(\n",
        "            name=\"first_sub_checklist_answer\",\n",
        "            confidence=0.5\n",
        "          )]\n",
        "        ))\n",
        "      ]\n",
        "    )]\n",
        "  )\n",
        ")\n",
        "\n",
        "nested_checklist_prediction_ndjson = {\n",
        "    \"name\": \"nested_checklist_question\",\n",
        "    \"answer\": [{\n",
        "        \"name\": \"first_checklist_answer\",\n",
        "        \"confidence\": 0.5,\n",
        "        \"classifications\": [{\n",
        "            \"name\": \"sub_checklist_question\",\n",
        "            \"answer\": {\n",
        "                \"name\": \"first_sub_checklist_answer\",\n",
        "                \"confidence\": 0.5\n",
        "            }\n",
        "        }]\n",
        "    }]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Classification: Checklist (multi-choice)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Python Annotations\n",
        "checklist_prediction = lb_types.ClassificationAnnotation(\n",
        "  name=\"checklist_question\",\n",
        "  value=lb_types.Checklist(\n",
        "      answer = [\n",
        "        lb_types.ClassificationAnswer(\n",
        "            name = \"first_checklist_answer\",\n",
        "            confidence=0.5\n",
        "        ),\n",
        "        lb_types.ClassificationAnswer(\n",
        "            name = \"second_checklist_answer\",\n",
        "            confidence=0.5\n",
        "        )\n",
        "      ]\n",
        "    )\n",
        " )\n",
        "\n",
        "# NDJSON\n",
        "checklist_prediction_ndjson = {\n",
        "  \"name\": \"checklist_question\",\n",
        "  \"answer\": [\n",
        "    {\"name\": \"first_checklist_answer\" , \"confidence\": 0.5},\n",
        "    {\"name\": \"second_checklist_answer\", \"confidence\": 0.5}\n",
        "  ]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Bounding Box"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Python Annotation\n",
        "bbox_prediction = lb_types.ObjectAnnotation(\n",
        "  name = \"bounding_box\",\n",
        "  confidence=0.5,\n",
        "  value=lb_types.Rectangle(\n",
        "        start=lb_types.Point(x=1690, y=977),  #  x = left, y = top\n",
        "        end=lb_types.Point(x=1915, y=1307),  # x= left + width , y = top + height\n",
        "    )\n",
        ")\n",
        "\n",
        "#NDJSON\n",
        "bbox_prediction_ndjson = {\n",
        "  \"name\": \"bounding_box\",\n",
        "  \"confidence\": 0.5,\n",
        "  \"bbox\": {\n",
        "          \"top\": 977,\n",
        "          \"left\": 1690,\n",
        "          \"height\": 330,\n",
        "          \"width\": 225\n",
        "      }\n",
        "}\n"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Bounding box with nested classification "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "####### Bounding box with nested classification #######\n",
        "bbox_with_radio_subclass_prediction = lb_types.ObjectAnnotation(\n",
        "    name=\"bbox_with_radio_subclass\",\n",
        "    confidence=0.5,\n",
        "    value=lb_types.Rectangle(\n",
        "        start=lb_types.Point(x=541, y=933),  #  x = left, y = top\n",
        "        end=lb_types.Point(x=871, y=1124),  # x= left + width , y = top + height\n",
        "    ),\n",
        "    classifications=[\n",
        "    \tlb_types.ClassificationAnnotation(\n",
        "        \tname=\"sub_radio_question\",\n",
        "      \t\tvalue=lb_types.Radio(answer=lb_types.ClassificationAnswer(name=\"first_sub_radio_answer\", confidence=0.5))\n",
        "    )\n",
        "  ]\n",
        ")\n",
        "\n",
        "\n",
        "## NDJSON\n",
        "bbox_with_radio_subclass_prediction_ndjson = {\n",
        "    \"name\": \"bbox_with_radio_subclass\",\n",
        "    \"confidence\": 0.5,\n",
        "    \"classifications\": [{\n",
        "        \"name\": \"sub_radio_question\",\n",
        "        \"answer\":\n",
        "            { \"name\":\"first_sub_radio_answer\", \"confidence\": 0.5}\n",
        "\n",
        "    }],\n",
        "    \"bbox\": {\n",
        "          \"top\": 933,\n",
        "          \"left\": 541,\n",
        "          \"height\": 191,\n",
        "          \"width\": 330\n",
        "        }\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Polygon"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Python Anotation\n",
        "polygon_prediction = lb_types.ObjectAnnotation(\n",
        "  name = \"polygon\",\n",
        "  confidence = 0.5,\n",
        "  value=lb_types.Polygon(\n",
        "        points=[lb_types.Point(x=1489.581,y=183.934), lb_types.Point(x=2278.306,y=256.885), lb_types.Point(x=2428.197,y=200.437), lb_types.Point(x=2560.0,y=335.419),\n",
        "                lb_types.Point(x=2557.386,y=503.165), lb_types.Point(x=2320.596,y=503.103), lb_types.Point(x=2156.083, y=628.943), lb_types.Point(x=2161.111,y=785.519),\n",
        "                lb_types.Point(x=2002.115, y=894.647), lb_types.Point(x=1838.456,y=877.874), lb_types.Point(x=1436.53,y=874.636), lb_types.Point(x=1411.403,y=758.579),\n",
        "                lb_types.Point(x=1353.853,y=751.74), lb_types.Point(x=1345.264, y=453.461), lb_types.Point(x=1426.011,y=421.129)]\n",
        "    ),\n",
        ")\n",
        "\n",
        "\n",
        "# NDJSON\n",
        "\n",
        "polygon_prediction_ndjson = {\n",
        "  \"name\": \"polygon\",\n",
        "  \"confidence\": 0.5,\n",
        "  \"polygon\": [\n",
        "    {\"x\": 1489.581, \"y\": 183.934},\n",
        "    {\"x\": 2278.306, \"y\": 256.885},\n",
        "    {\"x\": 2428.197, \"y\": 200.437},\n",
        "    {\"x\": 2560.0, \"y\": 335.419},\n",
        "    {\"x\": 2557.386, \"y\": 503.165},\n",
        "    {\"x\": 2320.596, \"y\": 503.103},\n",
        "    {\"x\": 2156.083, \"y\": 628.943},\n",
        "    {\"x\": 2161.111, \"y\": 785.519},\n",
        "    {\"x\": 2002.115, \"y\": 894.647},\n",
        "    {\"x\": 1838.456, \"y\": 877.874},\n",
        "    {\"x\": 1436.53, \"y\": 874.636},\n",
        "    {\"x\": 1411.403, \"y\": 758.579},\n",
        "    {\"x\": 1353.853, \"y\": 751.74},\n",
        "    {\"x\": 1345.264, \"y\": 453.461},\n",
        "    {\"x\": 1426.011, \"y\": 421.129},\n",
        "    {\"x\": 1489.581, \"y\": 183.934}\n",
        "  ]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Classification: Free-form text"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Python annotation\n",
        "text_annotation = lb_types.ClassificationAnnotation(\n",
        "  name=\"free_text\",\n",
        "  value=lb_types.Text(answer=\"sample text\", confidence=0.5)\n",
        ")\n",
        "\n",
        "# NDJSON\n",
        "text_annotation_ndjson = {\n",
        "  \"name\": \"free_text\",\n",
        "  \"answer\": \"sample text\",\n",
        "  \"confidence\": 0.5\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Segmentation mask"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "### Raster Segmentation (Byte string array)\n",
        "url = \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/raster_seg.png\"\n",
        "response = requests.get(url)\n",
        "\n",
        "mask_data = lb.types.MaskData(im_bytes=response.content) # You can also use \"url\" instead of img_bytes to pass the PNG mask url.\n",
        "mask_prediction = lb_types.ObjectAnnotation(\n",
        "    name=\"mask\",\n",
        "    value=lb_types.Mask(\n",
        "        mask=mask_data,\n",
        "        color=(255, 255, 255))\n",
        ")\n",
        "\n",
        "# NDJSON using instanceURI, bytes array is not fully supported.\n",
        "mask_prediction_ndjson = {\n",
        "  \"name\": \"mask\",\n",
        "  \"classifications\": [],\n",
        "  \"mask\": {\"instanceURI\": url,\n",
        "  \"colorRGB\": (255, 255, 255)}\n",
        "}\n",
        "\n"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Segmentation mask with nested classification"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "url_2 = \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/raster_seg_with_subclass.png\"\n",
        "response_2 = requests.get(url_2)\n",
        "mask_data_2 = lb_types.MaskData(im_bytes=response_2.content)\n",
        "\n",
        "# Python annotation\n",
        "mask_with_text_subclass_prediction = lb_types.ObjectAnnotation(\n",
        "  name = \"mask_with_text_subclass\", # must match your ontology feature\"s name\n",
        "  value=lb_types.Mask(\n",
        "    mask=mask_data_2,\n",
        "    color=(255, 255, 255)),\n",
        "  classifications=[\n",
        "    lb_types.ClassificationAnnotation(\n",
        "      name=\"sub_free_text\",\n",
        "      value=lb_types.Text(answer=\"free text answer\")\n",
        "    )]\n",
        ")\n",
        "\n",
        "# NDJSON using instanceURI, bytes array is not fully supported.\n",
        "mask_with_text_subclass_prediction_ndjson = {\n",
        "  \"name\": \"mask_with_text_subclass\",\n",
        "  \"mask\": {\"instanceURI\": url_2,\n",
        "  \"colorRGB\": (255, 255, 255)},\n",
        "  \"classifications\":[{\n",
        "    \"name\": \"sub_free_text\",\n",
        "    \"answer\": \"free text answer\"\n",
        "  }]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Point"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Python Annotation\n",
        "point_prediction = lb_types.ObjectAnnotation(\n",
        "  name = \"point\",\n",
        "  confidence=0.5,\n",
        "  value = lb_types.Point(x=1166.606, y=1441.768),\n",
        ")\n",
        "\n",
        "\n",
        "# NDJSON\n",
        "point_prediction_ndjson = {\n",
        "  \"name\": \"point\",\n",
        "  \"confidence\": 0.5,\n",
        "  \"classifications\": [],\n",
        "  \"point\": {\"x\": 1166.606, \"y\": 1441.768}\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Polyline"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Python Annotation\n",
        "\n",
        "polyline_prediction = lb_types.ObjectAnnotation(\n",
        "  name = \"polyline\",\n",
        "  confidence=0.5,\n",
        "  value=lb_types.Line(\n",
        "        points=[lb_types.Point(x=2534.353, y=249.471), lb_types.Point(x=2429.492, y=182.092), lb_types.Point(x=2294.322, y=221.962), lb_types.Point(x=2224.491, y=180.463), lb_types.Point(x=2136.123, y=204.716),\n",
        "                lb_types.Point(x=1712.247, y=173.949), lb_types.Point(x=1703.838, y=84.438), lb_types.Point(x=1579.772, y=82.61), lb_types.Point(x=1583.442, y=167.552),\n",
        "                lb_types.Point(x=1478.869, y=164.903), lb_types.Point(x=1418.941, y=318.149), lb_types.Point(x=1243.128, y=400.815), lb_types.Point(x=1022.067, y=319.007),\n",
        "                lb_types.Point(x=892.367, y=379.216), lb_types.Point(x=670.273, y=364.408), lb_types.Point(x=613.114, y=288.16), lb_types.Point(x=377.559, y=238.251),\n",
        "                lb_types.Point(x=368.087, y=185.064), lb_types.Point(x=246.557, y=167.286), lb_types.Point(x=236.648, y=285.61), lb_types.Point(x=90.929, y=326.412)]\n",
        "    ),\n",
        ")\n",
        "\n",
        "# NDJSON\n",
        "polyline_prediction_ndjson = {\n",
        "  \"name\": \"polyline\",\n",
        "  \"confidence\":0.5,\n",
        "  \"classifications\": [],\n",
        "  \"line\": [\n",
        "    {\"x\": 2534.353, \"y\": 249.471},\n",
        "    {\"x\": 2429.492, \"y\": 182.092},\n",
        "    {\"x\": 2294.322, \"y\": 221.962},\n",
        "    {\"x\": 2224.491, \"y\": 180.463},\n",
        "    {\"x\": 2136.123, \"y\": 204.716},\n",
        "    {\"x\": 1712.247, \"y\": 173.949},\n",
        "    {\"x\": 1703.838, \"y\": 84.438},\n",
        "    {\"x\": 1579.772, \"y\": 82.61},\n",
        "    {\"x\": 1583.442, \"y\": 167.552},\n",
        "    {\"x\": 1478.869, \"y\": 164.903},\n",
        "    {\"x\": 1418.941, \"y\": 318.149},\n",
        "    {\"x\": 1243.128, \"y\": 400.815},\n",
        "    {\"x\": 1022.067, \"y\": 319.007},\n",
        "    {\"x\": 892.367, \"y\": 379.216},\n",
        "    {\"x\": 670.273, \"y\": 364.408},\n",
        "    {\"x\": 613.114, \"y\": 288.16},\n",
        "    {\"x\": 377.559, \"y\": 238.251},\n",
        "    {\"x\": 368.087, \"y\": 185.064},\n",
        "    {\"x\": 246.557, \"y\": 167.286},\n",
        "    {\"x\": 236.648, \"y\": 285.61},\n",
        "    {\"x\": 90.929, \"y\": 326.412}\n",
        "  ]\n",
        "}\n"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 1: Import data rows into Catalog"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# send a sample image as batch to the project\n",
        "global_key = \"2560px-Kitano_Street_Kobe01s.jpeg\"\n",
        "test_img_url = {\n",
        "    \"row_data\": \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg\",\n",
        "    \"global_key\": global_key\n",
        "}\n",
        "dataset = client.create_dataset(name=\"image_prediction_demo\")\n",
        "task = dataset.create_data_rows([test_img_url])\n",
        "print(\"Errors:\",task.errors)\n",
        "print(\"Failed data rows:\", task.failed_data_rows)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 2: Create/select an Ontology for your model predictions\n",
        "Your project should have the correct ontology setup with all the tools and classifications supported for your annotations, and the tool names and classification instructions should match the name/instructions fields in your annotations to ensure the correct feature schemas are matched.\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "ontology_builder = lb.OntologyBuilder(\n",
        "  classifications=[ # List of Classification objects\n",
        "    lb.Classification(\n",
        "      class_type=lb.Classification.Type.RADIO,\n",
        "      name=\"radio_question\",\n",
        "      options=[\n",
        "        lb.Option(value=\"first_radio_answer\"),\n",
        "        lb.Option(value=\"second_radio_answer\")\n",
        "      ]\n",
        "    ),\n",
        "    lb.Classification(\n",
        "      class_type=lb.Classification.Type.CHECKLIST,\n",
        "      name=\"checklist_question\",\n",
        "      options=[\n",
        "        lb.Option(value=\"first_checklist_answer\"),\n",
        "        lb.Option(value=\"second_checklist_answer\")\n",
        "      ]\n",
        "    ),\n",
        "    lb.Classification(\n",
        "      class_type=lb.Classification.Type.TEXT,\n",
        "      name=\"free_text\"\n",
        "    ),\n",
        "    lb.Classification(\n",
        "        class_type=lb.Classification.Type.RADIO,\n",
        "        name=\"nested_radio_question\",\n",
        "        options=[\n",
        "            lb.Option(\"first_radio_answer\",\n",
        "                options=[\n",
        "                    lb.Classification(\n",
        "                        class_type=lb.Classification.Type.RADIO,\n",
        "                        name=\"sub_radio_question\",\n",
        "                        options=[lb.Option(\"first_sub_radio_answer\")]\n",
        "                    )\n",
        "                ]\n",
        "            )\n",
        "          ]\n",
        "        ),\n",
        "    lb.Classification(\n",
        "      class_type=lb.Classification.Type.CHECKLIST,\n",
        "      name=\"nested_checklist_question\",\n",
        "      options=[\n",
        "          lb.Option(\"first_checklist_answer\",\n",
        "            options=[\n",
        "              lb.Classification(\n",
        "                  class_type=lb.Classification.Type.CHECKLIST,\n",
        "                  name=\"sub_checklist_question\",\n",
        "                  options=[lb.Option(\"first_sub_checklist_answer\")]\n",
        "              )\n",
        "          ]\n",
        "        )\n",
        "      ]\n",
        "    ),\n",
        "  ],\n",
        "  tools=[ # List of tools\n",
        "    lb.Tool(\n",
        "      tool=lb.Tool.Type.BBOX,\n",
        "      name=\"bounding_box\"),\n",
        "    lb.Tool(\n",
        "      tool=lb.Tool.Type.BBOX,\n",
        "      name=\"bbox_with_radio_subclass\",\n",
        "      classifications=[\n",
        "            lb.Classification(\n",
        "                class_type=lb.Classification.Type.RADIO,\n",
        "                name=\"sub_radio_question\",\n",
        "                options=[\n",
        "                  lb.Option(value=\"first_sub_radio_answer\")\n",
        "                ]\n",
        "              ),\n",
        "        ]\n",
        "      ),\n",
        "    lb.Tool(\n",
        "      tool=lb.Tool.Type.POLYGON,\n",
        "      name=\"polygon\"),\n",
        "    lb.Tool(\n",
        "      tool=lb.Tool.Type.RASTER_SEGMENTATION,\n",
        "      name=\"mask\"),\n",
        "    lb.Tool(\n",
        "      tool=lb.Tool.Type.RASTER_SEGMENTATION,\n",
        "      name=\"mask_with_text_subclass\",\n",
        "      classifications=[\n",
        "          lb.Classification(\n",
        "              class_type=lb.Classification.Type.TEXT,\n",
        "              name=\"sub_free_text\")\n",
        "          ]\n",
        "      ),\n",
        " \t  lb.Tool(\n",
        "      tool=lb.Tool.Type.POINT,\n",
        "      name=\"point\"),\n",
        "    lb.Tool(\n",
        "      tool=lb.Tool.Type.LINE,\n",
        "      name=\"polyline\")]\n",
        ")\n",
        "\n",
        "ontology = client.create_ontology(\"Image Prediction Import Demo\",\n",
        "                                  ontology_builder.asdict(),\n",
        "                                  media_type=lb.MediaType.Image\n",
        "                                  )"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 3: Create a Model and Model Run"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# create Model\n",
        "model = client.create_model(name=\"image_model_run_\" + str(uuid.uuid4()),\n",
        "                            ontology_id=ontology.uid)\n",
        "# create Model Run\n",
        "model_run = model.create_model_run(\"iteration 1\")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 4: Send data rows to the Model Run"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "model_run.upsert_data_rows(global_keys=[global_key])"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 5. Create the predictions payload\n",
        "\n",
        "Create the prediction payload using the snippets of code in ***Supported Predictions*** section. \n",
        "\n",
        "The resulting label_ndjson should have exactly the same content for predictions that are supported by both"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Create a Label for predictions\n",
        "label_prediction = []\n",
        "label_prediction.append(lb_types.Label(\n",
        "    data=lb_types.ImageData(global_key=global_key),\n",
        "    annotations = [\n",
        "      radio_prediction,\n",
        "      nested_radio_prediction,\n",
        "      checklist_prediction,\n",
        "      nested_checklist_prediction,\n",
        "      bbox_prediction,\n",
        "      bbox_with_radio_subclass_prediction,\n",
        "      polyline_prediction,\n",
        "      polygon_prediction,\n",
        "      mask_prediction,\n",
        "      mask_with_text_subclass_prediction,\n",
        "      point_prediction,\n",
        "      text_annotation\n",
        "      ]\n",
        "  )\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "If using NDJSON:"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "label_prediction_ndjson = []\n",
        "\n",
        "for annot in [\n",
        "    radio_prediction_ndjson,\n",
        "    checklist_prediction_ndjson,\n",
        "    bbox_prediction_ndjson,\n",
        "    bbox_with_radio_subclass_prediction_ndjson,\n",
        "    polygon_prediction_ndjson,\n",
        "    mask_prediction_ndjson,\n",
        "    mask_with_text_subclass_prediction_ndjson,\n",
        "    point_prediction_ndjson,\n",
        "    polyline_prediction_ndjson,\n",
        "    text_annotation_ndjson,\n",
        "    nested_radio_prediction_ndjson,\n",
        "    nested_checklist_prediction_ndjson\n",
        "]:\n",
        "  annot.update({\n",
        "      \"dataRow\": {\"globalKey\": global_key}\n",
        "  })\n",
        "  label_prediction_ndjson.append(annot)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 6. Upload the predictions payload to the Model Run "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Upload the prediction label to the Model Run\n",
        "upload_job_prediction = model_run.add_predictions(\n",
        "    name=\"prediction_upload_job\"+str(uuid.uuid4()),\n",
        "    predictions=label_prediction)\n",
        "\n",
        "# Errors will appear for prediction uploads that failed.\n",
        "print(\"Errors:\",  upload_job_prediction.errors)\n",
        "print(\"Status of uploads: \", upload_job_prediction.statuses)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 7: Send annotations to a model run\n",
        "To visualize both annotations and predictions in the model run we will create a project with ground truth annotations. \n",
        "To send annotations to a Model Run, we must first import them into a project, create a label payload and then send them to the Model Run."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "##### 7.1. Create a labelbox project"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Create a Labelbox project\n",
        "project = client.create_project(name=\"Image Prediction Demo\",\n",
        "                                    media_type=lb.MediaType.Image)\n",
        "project.setup_editor(ontology)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### 7.2. Create a batch to send to the project "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "project.create_batch(\n",
        "  \"batch_predictions_demo\", # Each batch in a project must have a unique name\n",
        "  global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys\n",
        "  priority=5 # priority between 1(Highest) - 5(lowest)\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### 7.3 Create the annotations payload"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "########### Annotations ###########\n",
        "radio_annotation = lb_types.ClassificationAnnotation(\n",
        "    name=\"radio_question\",\n",
        "    value=lb_types.Radio(answer=lb_types.ClassificationAnswer(\n",
        "        name=\"second_radio_answer\"))\n",
        ")\n",
        "\n",
        "nested_radio_annotation = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_radio_question\",\n",
        "  value=lb_types.Radio(\n",
        "    answer=lb_types.ClassificationAnswer(\n",
        "      name=\"first_radio_answer\",\n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_radio_question\",\n",
        "          value=lb_types.Radio(\n",
        "            answer=lb_types.ClassificationAnswer(\n",
        "              name=\"first_sub_radio_answer\"\n",
        "            )\n",
        "          )\n",
        "        )\n",
        "      ]\n",
        "    )\n",
        "  )\n",
        ")\n",
        "\n",
        "nested_checklist_annotation = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_checklist_question\",\n",
        "  value=lb_types.Checklist(\n",
        "    answer=[lb_types.ClassificationAnswer(\n",
        "      name=\"first_checklist_answer\",\n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_checklist_question\",\n",
        "          value=lb_types.Checklist(\n",
        "            answer=[lb_types.ClassificationAnswer(\n",
        "            name=\"first_sub_checklist_answer\"\n",
        "          )]\n",
        "        ))\n",
        "      ]\n",
        "    )]\n",
        "  )\n",
        ")\n",
        "\n",
        "checklist_annotation = lb_types.ClassificationAnnotation(\n",
        "    name=\"checklist_question\",\n",
        "    value=lb_types.Checklist(answer=[\n",
        "        lb_types.ClassificationAnswer(name=\"first_checklist_answer\"),\n",
        "        lb_types.ClassificationAnswer(name=\"second_checklist_answer\")\n",
        "    ])\n",
        ")\n",
        "\n",
        "bbox_annotation = lb_types.ObjectAnnotation(\n",
        "  name = \"bounding_box\",\n",
        "  value=lb_types.Rectangle(\n",
        "        start=lb_types.Point(x=1690, y=977),  #  x = left, y = top\n",
        "        end=lb_types.Point(x=1915, y=1307),  # x= left + width , y = top + height\n",
        "    )\n",
        ")\n",
        "\n",
        "bbox_with_radio_subclass_annotation = lb_types.ObjectAnnotation(\n",
        "    name=\"bbox_with_radio_subclass\",\n",
        "    value=lb_types.Rectangle(\n",
        "        start=lb_types.Point(x=541, y=933),  #  x = left, y = top\n",
        "        end=lb_types.Point(x=871, y=1124),  # x= left + width , y = top + height\n",
        "    ),\n",
        "    classifications=[\n",
        "    \tlb_types.ClassificationAnnotation(\n",
        "        \tname=\"sub_radio_question\",\n",
        "      \t\tvalue=lb_types.Radio(answer=lb_types.ClassificationAnswer(name=\"first_sub_radio_answer\", confidence=0.5))\n",
        "    )\n",
        "  ]\n",
        ")\n",
        "\n",
        "polygon_annotation = lb_types.ObjectAnnotation(\n",
        "  name = \"polygon\",\n",
        "  value=lb_types.Polygon(\n",
        "        points=[lb_types.Point(x=1489.581,y=183.934), lb_types.Point(x=2278.306,y=256.885), lb_types.Point(x=2428.197,y=200.437), lb_types.Point(x=2560.0,y=335.419),\n",
        "                lb_types.Point(x=2557.386,y=503.165), lb_types.Point(x=2320.596,y=503.103), lb_types.Point(x=2156.083, y=628.943), lb_types.Point(x=2161.111,y=785.519),\n",
        "                lb_types.Point(x=2002.115, y=894.647), lb_types.Point(x=1838.456,y=877.874), lb_types.Point(x=1436.53,y=874.636), lb_types.Point(x=1411.403,y=758.579),\n",
        "                lb_types.Point(x=1353.853,y=751.74), lb_types.Point(x=1345.264, y=453.461), lb_types.Point(x=1426.011,y=421.129)]\n",
        "    ),\n",
        ")\n",
        "\n",
        "text_annotation = lb_types.ClassificationAnnotation(\n",
        "    name=\"free_text\",\n",
        "    value=lb_types.Text(answer=\"sample text\")\n",
        ")\n",
        "\n",
        "mask_annotation = lb_types.ObjectAnnotation(\n",
        "    name=\"mask\",\n",
        "    value=lb_types.Mask(\n",
        "        mask=mask_data,\n",
        "        color=(255, 255, 255))\n",
        ")\n",
        "\n",
        "mask_with_text_subclass_annotation = lb_types.ObjectAnnotation(\n",
        "  name = \"mask_with_text_subclass\", # must match your ontology feature\"s name\n",
        "  value=lb_types.Mask(\n",
        "    mask=mask_data_2,\n",
        "    color=(255, 255, 255)),\n",
        "  classifications=[\n",
        "    lb_types.ClassificationAnnotation(\n",
        "      name=\"sub_free_text\",\n",
        "      value=lb_types.Text(answer=\"free text answer\")\n",
        "    )]\n",
        ")\n",
        "\n",
        "point_annotation = lb_types.ObjectAnnotation(\n",
        "  name = \"point\",\n",
        "  value = lb_types.Point(x=1166.606, y=1441.768),\n",
        ")\n",
        "\n",
        "polyline_annotation = lb_types.ObjectAnnotation(\n",
        "  name = \"polyline\",\n",
        "  value=lb_types.Line(\n",
        "        points=[lb_types.Point(x=2534.353, y=249.471), lb_types.Point(x=2429.492, y=182.092), lb_types.Point(x=2294.322, y=221.962), lb_types.Point(x=2224.491, y=180.463), lb_types.Point(x=2136.123, y=204.716),\n",
        "                lb_types.Point(x=1712.247, y=173.949), lb_types.Point(x=1703.838, y=84.438), lb_types.Point(x=1579.772, y=82.61), lb_types.Point(x=1583.442, y=167.552),\n",
        "                lb_types.Point(x=1478.869, y=164.903), lb_types.Point(x=1418.941, y=318.149), lb_types.Point(x=1243.128, y=400.815), lb_types.Point(x=1022.067, y=319.007),\n",
        "                lb_types.Point(x=892.367, y=379.216), lb_types.Point(x=670.273, y=364.408), lb_types.Point(x=613.114, y=288.16), lb_types.Point(x=377.559, y=238.251),\n",
        "                lb_types.Point(x=368.087, y=185.064), lb_types.Point(x=246.557, y=167.286), lb_types.Point(x=236.648, y=285.61), lb_types.Point(x=90.929, y=326.412)]\n",
        "    )\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### 7.4. Create the label object"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Create a Label object by identifying the applicable data row in Labelbox and providing a list of annotations\n",
        "label = []\n",
        "annotations = [\n",
        "    radio_annotation,\n",
        "    nested_radio_annotation,\n",
        "    checklist_annotation,\n",
        "    nested_checklist_annotation,\n",
        "    text_annotation,\n",
        "    bbox_annotation,\n",
        "    bbox_with_radio_subclass_annotation,\n",
        "    polygon_annotation,\n",
        "    mask_annotation,\n",
        "    mask_with_text_subclass_annotation,\n",
        "    point_annotation,\n",
        "    polyline_annotation\n",
        "]\n",
        "label.append(\n",
        "    lb_types.Label(\n",
        "        data=lb_types.ImageData(global_key=global_key),\n",
        "        annotations=annotations))"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### 7.5. Upload annotations to the project using Label Import"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "upload_job_annotation = lb.LabelImport.create_from_objects(\n",
        "    client = client,\n",
        "    project_id = project.uid,\n",
        "    name=\"annotation_import_\" + str(uuid.uuid4()),\n",
        "    labels=label)\n",
        "\n",
        "upload_job_annotation.wait_until_done()\n",
        "# Errors will appear for annotation uploads that failed.\n",
        "print(\"Errors:\", upload_job_annotation.errors)\n",
        "print(\"Status of uploads: \", upload_job_annotation.statuses)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### 7.6 Send the annotations to the Model Run"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# get the annotations from the project and add them to the model\n",
        "model_run.upsert_labels(project_id=project.uid)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Optional deletions for cleanup \n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# project.delete()\n",
        "# dataset.delete()"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    }
  ]
}